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Abstract: Similar to what happens between humans in the real world, in open multi-agent systems distributed over the 
Internet, such as online social networks or wiki technologies, agents often form coalitions by agreeing to act as 
a whole in order to achieve certain common goals. However, agent coalitions are not always a desirable feature 
of a system, as malicious or corrupt agents may collaborate in order to subvert or attack the system. In this 
paper, we consider the problem of hidden coalitions, whose existence and the purposes they aim to achieve are 
not known to the system, and which carry out so-called underhand attacks. We give a first approach to hidden 
coalitions by introducing a deterministic method that blocks the actions of potentially dangerous agents, i.e. 
possibly belonging to such coalitions. We also give a non-deterministic version of this method that blocks 
the smallest set of potentially dangerous agents. We calculate the computational cost of our two blocking 
methods, and prove their soundness and completeness. 



1 Introduction 

1.1 Context and motivation 

Similar to what happens between humans 
in the real world, in open multi-agent sys- 
tems (Davidsson, 2001) distributed over the Internet, 
such as online social networks or wiki technologies, 
agents often form coalitions by agreeing to act 
as a whole in order to achieve certain common 
goals. For instance, agents may wish to collab- 
orate in order to jointly create and use a group 
cryptographic key for ensuring the confidentiality 
and/or integrity of information shared within the 
group, e.g. (Raf aeli and Hutchison, 2003] ), or to 
partake in a mix network or some other anonymous 
remailer to achieve unobservability of communi- 
cations, e.g. (Chaum, 1981 ), or to create secret 
interest groups within online social networks, 
e.g. (Sorniotti and Molva, 2010). However, agent 
coalitions are not always a desirable feature of a 
system, as malicious or corrupt agents may collab- 
orate in order to subvert or attack the system. For 
instance, such agents may collaborate to attack the 



information in transit over different channels in a web 
service architecture or in a distributed wired and/or 
wireless computer network, e.g. (Wiehler, 2004}, or 
they might forge and spread false information within 
the system, e.g. ( |Hahn et al., 2007| ). 

In order to be able to rigorously formalize 
and reason about such positive and negative 
properties of agent coalitions, and thereby allow 
for the prevention or, at least, the identifica- 
tion of the entailed vulnerabilities, a number 
of different formal approach es have been re- 
cently proposed, such as ( Agotnes et al., 2007 



Agotnes et al., 2008 



lAluret al., 1998b 



van der Hoek et al., 2005 \ Oravec and Fogel, 2006 



Pauly,2001| 



Troquard et al., 2009 



|van der Hoek a nd Wooldridg e, 2005 1 
|Wooldridge and Dunne, 2004) . 

In this paper, we consider the problem of hidden 
coalitions: a coalition is hidden in a system when 
its existence and the purposes it aims to achieve are 
not known to the system. Hidden coalitions carry 
out underhand attacks, a term that we borrow from 
military terminology. These attacks are particularly 
subtle since the agents that perform them are not out- 



siders but rather members of the system whose se- 
curity properties are posed under threat. Moreover, 
the mere suspect that a group of individuals act as a 
whole is typically insufficient to come to a decision 
about their permanence as members of the system; 
this, of course, depends also on the nature of the sys- 
tem and the information it contains, since in the pres- 
ence of highly security-sensitive information, systems 
may anyway opt for the exclusion of all suspected 
agents. However, in general, systems, and even more 
so open ones, will want to adopt a less restrictive pol- 
icy, excluding only those agents whose malice has in- 
deed been proved. Therefore, the defense against un- 
derhand attacks by hidden coalitions is a fundamental 
but complex matter. 

Problems of a similar kind have been studied, for 
instance, in Game Theory ( Aum anri and Hart, 1994[ 
Pauly and Parikh, 2003 ) in relation to the nature of 
collaboration and competition, and from the view- 
point of modeling group formation under the con- 
straints of possible given goals. However, underhand 
attacks by hidden coalitions pose security problems 
that cannot be dealt with such traditional means. Nor 
can they solved by a simple, monotonic, approach 



based on Coalition Logic(s) ( Agotnes et al., 2008 



Oravec and Fogel, 2006 



Pauly, 2001 



van der Hoek and Wooldridg e, 2005| l, which is 



currently one of the most successful formalisms for 
reasoning about coalitions. 

To illustrate all this further, consider the following 
concrete example from an online social network such 
as Facebook, where abuse, misuse or compromise of 
an account can be reported to the system administra- 
tion. In particular, a group of agents (in this case, 
Facebook users) can report a fake profile: 

You can report a profile that violates Facebook's 
Statement of Rights and Responsibilities by 
clicking the "Report/Block this Person" link in 
the bottom left column of the profile, select- 
ing "Fake profile" as the reason, and adding 
the appropriate information. [...] (Excerpt from 
Ihttp:// www. facebook. com/help/ ? sear ch=f ake) 

The administrator of the system gives an ultimatum to 
the agent that uses the reported profile and then may, 
eventually, close it. An underhand coalition can ex- 
ploit this report mechanism to attack an agent who 
possesses a "lawful" original profile: at first they cre- 
ate a fake profile with personal information and pho- 
tos of the agent under attack, and then they become 
friends of her. After that, they report the original pro- 
file so that the administrator closes it. The report is 
a lawful action, and by creating the new profile and 
having a big enough number of agents who report the 
same profile no suspicion about the hidden coalition 
is raised, so that the attack succeeds. 



1.2 Contributions 

A formalism to define and reason about such hidden 
coalitions is thus needed. Indeed, Coalition Logic al- 
lows one to define coalitions that are explicit (i.e. not 
hidden) and is characterized by monotonic permis- 
sions to act in groups and individually. What is miss- 
ing, however, is the notion of hidden coalition and a 
method to block the underhand attacks such coalitions 
carry out. The idea underlying our approach is to cir- 
cumscribe the problem in algebraic terms, by defining 
a system that can be represented by a coalition logic, 
and then activate a non-monotonic control on the sys- 
tem itself to block the underhand attacks that hidden 
coalitions are attempting to carry out. 

More specifically, we consider multi-agent sys- 
tems whose security properties depend on the val- 
ues of sets of logical formulas of propositional logic, 
which we call the critical (or security) formulas of the 
systems: for concreteness, we say that a system is se- 
cure if all the critical formulas are false, and is thus 
insecure if one or more critical formula is true. (Of 
course, we could also invert the definition and con- 
sider a system secure when all critical formulas are 
true.) The system agents control the critical formu- 
las in that they control the propositional variables that 
formulas are built from: we assume that every vari- 
able of the system is controlled by an agent, where 
the variables controlled by an agent are controlled 
just by that agent without interference by any other 
agent. The actions performed by each agent consist 
thus in changing some of the truth values of the vari- 
ables assigned to that agent, which means that the 
values of the critical formulas can change due to ac- 
tions performed by the agents, including in particular 
malicious insider agents who form hidden coalitions 
to attack the system by making critical formulas be- 
come true. Returning to the Facebook example, this 
is exactly what happens when agents report the origi- 
nal profile as fake by setting the flag (clicking on the 
link)Q 

At each instant of time, agents ask the system to 
carry out the actions they wish to perform, i.e. chang- 
ing the truth value of the variables they control, and 
the system has to decide whether to allow such ac- 
tions, but without knowing of the existence of possi- 
ble hidden coalitions and thus at the risk of the system 
becoming insecure. To block such attacks, we for- 



'in this paper, we do not consider how the administra- 
tor decides to close the profile, nor do we consider in de- 
tail the non-monotonic aspects of how agents enter/exit/are 
banned from a system or enter/exit a hidden coalition, or 
how members of a hidden coalition synchronize/organize 
their actions. All this will be subject of future work. 



malize here a deterministic blocking method, imple- 
mented by a greedy algorithm, which blocks the ac- 
tions of potentially dangerous agents. We prove that 
this method is sound and complete, in that it does not 
allow a system to go in an insecure state when it starts 
from a secure state and it ensures that every secure 
state can be reached from any secure state. However, 
this algorithm is not optimal as it does not block the 
smallest set of potentially dangerous agents. 

We thus introduce also a non-deterministic block- 
ing method, which we obtain by extending the deter- 
ministic method with an oracle to determine the min- 
imum set of agents to block so to ensure the secu- 
rity of the system. We show that the soundness and 
completeness result extends to this non-deterministic 
method as well. 

We also calculate the computational cost of our 
two blocking methods. This computational analysis 
is completed by determining upper bound results for 
the problem of finding a set of agents to be blocked 
so to prevent system transitions into insecure states, 
and the problem of finding an optimal set of agents 
satisfying the above condition. 

1.3 Organization of the paper 

In ^2] we introduce our approach to the problem of 
blocking underhand attacks by hidden coalitions. 5j3] 
and §|4] respectively introduce our deterministic and 
non-deterministic blocking methods, giving concrete 
examples for their application. In §|5] we study the 
computational aspects of these two methods, calcu- 
lating in particular their computational cost, and show 
that they are both sound and complete. Finally, in §|6] 
we summarize our main results, discuss related work 
and sketch future work. 

2 An approach to the problem of 
blocking underhand attacks 

We introduce our approach to the problem of 
blocking underhand attacks. We also recall 

some basic notions and, in particular, the rele- 
vant notions of the Coalition Logic of Proposi- 



tional Control CL-PC (Oravec and Fogel, 2006 



van der Hoek and Wooldridge, 2005 ), which pro- 
vides a starting point for our approach. 

2.1 Syntax 

We consider multi-agent systems S that are described 
by a set of critical (or security) formulas <S> and by a 
temporal sequence a : T — > ®(<P), with T the tempo- 
ral axis and ©(<!>) the propositional assignment in the 



set of formulas. In this work, we focus only on the 
formulas in <t>, which represent the security-critical 
characteristics of a system (which depend on the ap- 
plication and which we thus do not describe further 
here, as our approach is independent of the particular 
application). We say that a system is secure if all the 
critical formulas are false, and it becomes insecure if 
one or more (]) G 4> becomes true0 Hence, the state of 
a system is defined by the value of the propositional 
variables that occur in the critical formulas of <t>. 

The agents of a system S control the set <t> and 
hence the state of S- We require that there is no 
formula in our systems that cannot change its truth 
value. Moreover, the distribution of the variables to 
the agents should be such that one formula cannot 
be controlled by one single agent, but rather differ- 
ent agents control one formula, and every formula is 
controlled by some agents. In particular, for a set Ag 
of system agents: 

• every variable of the system is controlled by an 
agent a € Ag, and 

• the variables controlled by an agent are controlled 
just by that agent without interference by any 
other agent. 

The actions performed by each agent a £ Ag 
are thus the changing of the truth values 
of the variables assigned to a. The agents 
we consider are intelligent agents in the 
sense of ([Wooldridge and Jennings, 1995; 



Wooldridge and Jennings, 1998 1: they are au- 
tonomous, have social ability, reactivity and pro- 
activeness, and have mental attitudes, i.e. states that 
classify the relation of the agents and the cognitive 
environment. In our approach, we consider intelligent 
agents but do not make specific assumptions about 
their mental attitudes, except for their collaborative 
attitudes that constitute a threat to (the security of) 
the system^ 

In Game Theory ( |van der Hoek et al., 2005| l, 
strategies are often associated with a preference 
relation for each agent that indicates which output the 
agent is going to select in presence of alternatives. 
In our approach, agents change the value of "their" 
variables according to their strategies and create 
coalitions with other agents so to be more expressive: 
by collaborating, agents can change the values 
of different variables and thus, ultimately, of the 



2 As we remarked above, we could also invert this and 
call a system secure when all critical formulas are true; we 
would then just need to modify our methods accordingly. 

3 An extension of the work presented here with a detailed 
formalization of the mental and collaborative attitudes of 
the agents will be subject of future work. 



critical formulas that comprise such variables. The 
novelty in this work is that we don't deal just with 
coalitions that are known by the system but also with 
hidden coalitions, whose existence and purposes are 
unknown by the system. 

Let us now formalize the language of our ap- 
proach. Following CL-PC, given a set Ag of agents, a 
set Vars of propositional variables, the usual operators 
-i and V of classic propositional logic, and the coop- 
eration mode O, we consider formulas built using the 
following grammar: 

4>::=T|p|-4>|<|>V<|>|Oc<ti 

where p G Vars, C C Ag, and Oc§ is a cooperation 
formula. Slightly abusing notation, we denote with 
Vars(§) the set of propositional variables that occur in 
(j) and with Ag(§) the agents that control the variables 
in Vars(ty). Oc<|> expresses that the coalition C has 
the contingent ability to achieve (j); this means that the 
members of C control some variables of (j) and have 
choices for (j) such that if they make these choices and 
nothing else changes, then (j) will be true. 

2.2 Semantics 

A model is a tuple 

M = (Ag,Vars,Vars\i,...,Vars\ n ,Q) , 

where: 

• Ag= {l,...,n}is a finite, non-empty set of agents; 

• Vars = {p,q,...} is a finite, non-empty set of 
propositional variables; 

• Vars\\,...,Vars\ n is a partition of Vars among the 
members of Ag, with the intended interpretation 
that Vara | is the subset of Vars representing those 
variables under the control of agent ;' G Ag; 

• 8 : Vars — > {T,_L} is a propositional valuation 
function that determines the truth value of each 
propositional variable. 

Since Vars\ i, Vars\ n is a partition of Vars, we have: 

1. Vars = Vars\\ U ... U Vars\ n i.e. every variable is 
controlled by some agent; 

2. Vars\i fl Vars\j = for i^je Ag, i.e. no variable 
is controlled by more than one agent. 

We denote with Vfors|c the variables controlled by 
the agents that are part of the coalition C C Ag. 
Given a model M = {Ag,Vars,Vars\\, ...,Vars\ n ,B) 
and a coalition C, a C-valuation function is 8c : 
Vara|c — > {T,_L}. Valuations 8 extend from vari- 
ables to formulas in the usual way and for a model 
5W = (Ag,Vars,Vars\i,...,Vars\ n ,Q) we write M |= (j) 
if 8((])) = T. We write |= <|> if M |= for all M . 



2.3 Secure and insecure systems 

All the semantic notions introduced above actually 
depend on the current time, and we will thus decorate 
them with a superscript - s ' denoting the system state 
at time t, e.g. 8 s ' and \= s < . Time is discrete and natu- 
ral, and is defined with a non empty set of time points 
T and a transitive and irrefiexive relation -< such that 
t < u means that t comes before u for t,u G T. In 
our case, since t,t + 1 G T it follows naturally that 
t^t + l. 

The passing of time is regulated by a general 
clock, which ensures that the system can execute a 
definite number of actions in an instant of time: at ev- 
ery clock of time, the system changes its state, which 
is thus defined by the actions that the system executes. 
Even if there are no actions to execute, the system 
changes its state from S t to S t +i, which in this case 
are equal. 

We assume that each system S starts, at time to, 
from a secure state So, i.e. a state in which all the 
critical formulas of <t> are false, so that none of the 
features of the system is violated. In general: 

S is secure at a state S t iff ^= s ' (j) for all (j) G <t> 

and 

S is secure iff ^= St § for all (j) G 4> and all S t 

At time t, the system is in state S t and goes to state 
S t+ i and executes all the actions of the agents that 
want to change the value of their variables. Denoting 
with r r+ i the set of actions that the agents want to 
execute at the time instant t, we can write 

S t ^4 S,+i . 

and the aim of our approach is to guarantee that each 
reachable state S t+ i is secure, where the differences 
between S t and S t +i are in their respective ©. 

Since a coalition can change the value of the vari- 
ables it controls, it can attempt to change the value of 
a critical formula to true; formally, for a coalition C 
and a formula (j) if Oc<|) is true then it means that C can 
make (j) true and thus the system insecure, which we 
can write by negating the above definition or alterna- 
tively, and basically equivalently, as: 

S is insecure at a state S t iff \= s < Oc§ for some 
C C Ag and some (j) G <t> 

To help the control of the system (but without loss 
of generality), we can create a filter for the actions 
that imposes a limit on the number of the actions that 
can be executed in an instant of time. This can de- 
crease the performance of the system, so we need a 
trade-off between control and performance. 



Algorithm 1 A greedy, deterministic block- 
ing METHOD 



1: Simulate(T t+ i) = [<&'&']; 

2: while (4>' ^ 0) do 

3: Create the matrix with <P' and 9L 1 ; 

4: Vfl; £^':a;-> c,-,Cj = count (0,-); 

5: Quicksort (c i,.. c j) = (c v ,...); 

6: ® = 3 U a A ; {where a x is the agent associ- 
ated to c T , that is the maximum counter of the 
marked cells} 

7: 5/mw/afe(r f+ i\r| Ul .) = [& j?']; 

8: end while 



3 A deterministic blocking method 

Our aim is to introduce a method that guarantees the 
security of the system, which amounts to blocking the 
actions of hidden coalitions. Indeed, in the case of 
"normal" coalitions, the property Oc§ allows us to 
list the actions of the agents in C, while if the coali- 
tion is hidden then we cannot block any action as we 
cannot directly identify the participants of a coalition 
we do not even know to exist. Since the actions of 
participants of hidden coalitions are not predictable, 
we cannot oppose these coalitions using O, so we in- 
troduce a method that disregards the existence of this 
property. 

Our (main) method for the protection of the sys- 
tem is a blocking method based on the greedy Algo- 
rithm [T] the agents make a request to the system for 
the actions T t +i they wish to execute at time f, and the 
system then simulates (via a method Simulate we as- 
sume to exist) the actions in order to control whether 
the system after the execution of the actions is still 
secure or not. The simulation says if the system can 
proceed with the execution of the actions or not, in 
which case it is given a list of the formulas <t>' that 
became true along with the set of agents a ' that made 
them become true. 

If the simulation says that the system can go in an 
insecure state, the blocking method constructs a ma- 
trix: in every column of the matrix there is one of the 
agents given by the simulation and in every row there 
is one of the formulas that became true during the sim- 
ulation. We mark each cell that has as coordinates 
the agent that has variables in that formula, and then 
we eliminate the column that has more marked cells 

4 It would be more efficient to consider only the vari- 
ables of the formulas that become true, but if we take only 
these variables, we cannot prevent long-term strategies of 
hidden coalitions, consisting in the progressive reduction of 
the number of steps needed for making a security formula 



The corresponding agent is not eliminated, rather he is 
just blocked and his actions are not executed (by sub- 
tracting T\ ax ): the "dangerous" agents found in this 
way are put in a set <B of blocked agents. The simula- 
tion is called again and so on, until the output of the 
simulation is an empty set of formulas, which means 
that by executing the remaining actions the system 
does not go in an insecure state. It is important to 
note that this method does not prevent the creation of 
hidden coalitions but can guarantee the system secu- 
rity from the attacks made by these coalitions. 

The most important property of Algorithm Q] is 
that it never brings the system in an insecure state, 
as it blocks the actions of agents that can make the 
system insecure. We do not commit to a specific way 
that the blocking is actually done, as it depends on 
the particular observed systems and on the particular 
goals. For instance: 

• Block the agent from changing the value of his 
variables until a precise instant of time. During 
this period, his variables are left unchanged or are 
controlled by the superuser/system administrator. 

• Block the agent for an interval of time, which can 
be a default value or can be chosen in a random 
way, e.g. so that a hidden coalition doesn't know 
when the agent can be active and thus cannot or- 
ganize another attack. 

• Block the agent and remove his actions for that in- 
stant of time. At the next instant, the agent has the 
possibility to ask for his actions to be executed. 

• Leave the variables unchanged, without making 
known to the agent if the value of the variables 
has been changed or not. This method can be 
improved by blocking the agent if he attempts to 
change the truth value of those variables again. 

Other, more complex, blocking strategies can of 
course be given, e.g. by combining some the above. 

Note also that, depending on the system consid- 
ered, it could be that not all the requests for execution 
can be satisfied: the maximum number n of actions 
that can be executed in an instant of time can be cho- 
sen in different ways, with respect to the characteris- 
tics of the system. Here, we choose n to be the car- 
dinality |<t>| of the critical formulas. The order used 
for taking these actions and executing them respects a 
FIFO queue, so the first n actions are executed. 

Example 1 As a concrete example of the application 
of the blocking method, consider a system S defined 

true. An optimization of the choice of variables to be con- 
sidered in order to reduce the effectiveness of such long- 
term strategies will be subject of future work. 



by the critical formulas 

01 = vi A V2 A (-1V3 V V5 V -1V4) 
<|>2 = (-1V5V-1V3) A^v 6 

03 = V 7 A(-iV8V-iV 6 ) 

04 = (v'8 V V'5 V -1V9) A V'2 A V'l 

so that number of the action to be executed in an 
instant of time is n = 4 ( the cardinality of the set 
of critical formulas that define the system), and let 
Ag = {01,02, 0-5,04, a$} and At — {v\, ...,V9}. Fur- 
ther, consider the following distribution of the vari- 
ables to the agents: 



a\ 


= {l'l,V 7 ,Vg} 


«2 


= M 


«3 


= {V2,V 6 } 


«4 


= {V4,V 5 } 


«5 


= {v 9 } 



Let us assume that the state S t at time t is 

e Sf (vi) = e s '(v 5 ) = ± 

Q s > ( V2 ) = 9 s ' (v 3 ) = 6 s ' (v 4 ) = 9 s ' (v 6 ) = 9 s ' (v 7 ) 
= 9 s '(v 8 )=9 s '(v 9 ) = T 

and that we have the following actions T t+ i to be ex- 
ecuted at time t in the FIFO queue: 

9 5f +'(vi) 4 1 T, 

9 5 '+>(v 3 ) <-n ±, 

9 5f +'(v 4 ) ^ 1 JL, 

9 5 '+'(v 6 ) <h J_, 

77iaf w, vi should be set to T of sfafe S f +i, one/ 50 
on. The algorithm simulates the first n = 4 actions, so 
that 4>' = {(j)i,(j)2,03,04} ant/ = {01,02,03,04}, 
and the matrix of Table Q] is constructed, which the 
algorithm sorts by the highest counter to produce the 
matrix in Table [2] 03 is thus put into H . The simu- 
lation takes place again, taking into account that we 
have blocked the value of the variables controlled by 
03 at the truth value of the instant of time t. The 
simulation gives as result the set <t>' = {(j)i,(|)4} e 
Si' = {(j)i ,(j)2,04}. The matrix of Table\3\is created, 
which is already ordered ( so the sorting will return the 
same matrix). So, we put in H the agent a\, block its 
actions and make the simulation with the remaining 
actions. This simulation gives = 0, and thus the 
remaining actions can be executed without any risk 
for the system S. 



Table 1: Matrix constructed by the blocking algorithm for 
Example Q] 





a\ 


a 2 


a 3 


a A 


01 


X 


X 


X 


X 


02 




X 


X 


X 


03 


X 




X 




04 


X 




X 


X 



Table 2: Matrix of Table [T] sorted in a decreasing order of 
counters. 





«3 


a\ 


a 4 


«2 


0i 


X 


X 


X 


X 


02 


X 




X 


X 


03 


X 


X 






04 


X 


X 


X 





4 A non-deterministic blocking 
method 

As we will see in ^5] the above deterministic block- 
ing method based on a greedy algorithm is sound and 
complete. However, this algorithm is not optimal as it 
cannot block the smallest set of potentially danger- 
ous agents. We now introduce a non-deterministic 
method, which can be used for identifying optimal 
solutions. The method, which is implemented in Al- 
gorithms|2]and[3j is obtained by introducing an oracle 
(to determine the minimum set of agents to block so 
to ensure the security of the system) within the de- 
terministic version, which makes the soundness and 
completeness results directly applicable to the non- 
deterministic version as well. 

The idea is that the result given by the simulation 
is passed to the method ScanOracle, which creates all 
the subsets of the given set A 1 with cardinality \a' — 1 
and finds the subsets with the maximum number of 
critical formulas that remain false, using the simula- 
tion. The simulation of all the subsets is done in par- 
allel; the ScanOracle is the non-deterministic part of 
our algorithm. The result is passed to the main algo- 
rithm: if we find a subset of agents such that when 
executing their actions all critical formulas are false, 
then we have finished and we block the remaining 
agents that are not part of this subset; if not all the 
critical formulas remain false the result is passed re- 
cursively to ScanOracle until it is given a set of agents 
such that all the critical formulas stay false when sim- 
ulating their actions. The rest of the agents in A 1 that 
are not part of the given subset are blocked. Using 
this method, we can have different best solutions but 
we choose one in a random way, where with "best so- 



Table 3: Matrix of Table [2] after the block of agent 03. 





«1 


«2 


a 4 


♦1 


X 


X 




<t>4 


X 




X 



Algorithm 2 A non-deterministic blocking 

METHOD 

1: Simulate{T„) = 

2: I = A';j = 0; 

3: while ^ & / ^ & ./' < |^t'|) do 

4: /' = ScanOracle(I); 

5: For a random /, G /' 

6: if I {(j); I y= (j)/ at the current state } | = \<$>' | then 

7: 7 = 0; 

8: else 

9: /=/'; 

10: end if 

11: j + + 

12: end while 

13: Choose a subset I, e I' and put Jl'\Ii in <B 



Algorithm 3 ScanOracle 



1: Generate the subset of I with cardinality |/| — 1 
2: Execute the simulation in parallel for each subset 

/,-, where i£ {1,...,|/|} 
3: Take the /,■ with the maximum number of {(]),■ | ^ 

(j), } and put them in /' 
4: I' =l'UIi 

5: Eliminate the duplicates in /' 
6: Return/' 



lutions" we mean sets that have the same cardinality 
and are the biggest sets that make the critical formulas 
stay false, so that we block the smallest set of agents 
that make the critical formulas true. 

Example 2 As a concrete example of the applica- 
tion of Algorithm^ (and of Algorithm^, consider 
again the system of Example [7] with the same data. 
The simulation of the first 4 actions yields again 
A 1 = {01,02,03,04}, which is passed to ScanOracle, 
which in turn creates the subsets: 

h = {01,02,03} 

h = {01,03,04} 

13 = {01,02,04} 

14 = {02,03,04} 

The oracle takes these subsets and gives as results at 



the current state 
h ■ 

h: 
h- 
l 4 : 



I=4>1: 
Ml: 
Ml: 
Ml: 



M2, 
M2, 
M2, 
M2, 



M3- 

M3: 
M3: 
M3- 



M4- 
M4- 
M4- 

M 4 . 



The two subsets with maximum number of false criti- 
cal formulas are I3 and I4, so I' = I3 UI4. Note that, 
since I3 and I 4 have the same number of false formu- 
las, it is enough to test just one of them to see if all the 
formulas are false or not; in this case, we have just 
two formulas. The ScanOracle is then called again 
with I 1 = I3UI4 and it yields the subsets 
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= {01,02} 


h 


= {01,04} 


h 


= {02,04} 


h 


= {02,03} 
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= {02,04} 


ho 


= {03,04} 



and thus the following results 
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Then /' = I-j Dig = h as these two subsets are identi- 
cal. Using I , all the critical formulas are false, so it is 
the maximum subset of agents with which the system 
is secure. Hence, we block the remaining agents in Si 1 , 
which is the minimum set of agents for the blocking of 
which the system remains secure: 

{ai,a2,a3,a4}\{a 2 ,a 4 } = {01,03}. 



5 Computational cost, Soundness 
and Completeness 

In this section, we calculate the computational cost of 
both the blocking methods we have given, and then 
show that the greedy deterministic method is sound 
and complete (which implies the same for the non- 
deterministic method). Recall that the maximum 
number n of actions that can be executed in an instant 
of time corresponds to the cardinality of the formulas 
in <t>. So, in the worst case, at each instant of time, 
there are n different agents that want to change the 
value of n different variables. 

Theorem 1 The computational cost of the greedy 
blocking method, Algorithm^ is o(« 3 ). 



Proof The simulation costs n because if we have n 
variables and all of them are part of all the formulas, 
we need to do n 2 assignments. The while cycle needs 
to be executed at worst n times as it blocks an agent 
per cycle and we may need to block all the n agent. 
The cost of a single while cycle is 3n 2 due to the sum 
of: the cost of the matrix creation, which is n 2 , the 
cost of the association of the counter, which is n 2 , and 
the cost of the quicksort algorithm, which we take to 
be n 2 , instead of nlogn (as the simulation has an n 2 
complexity). So, the total cost of the algorithm is n 2 + 
[n x (3n 2 + n 2 )) = o(n 3 ). " □ 

Theorem 2 The computational cost of the non- 
deterministic blocking method, Algorithm^ is o(« 3 ). 

Proof Also this algorithm uses the simulation, which 
costs n 2 . In the worst case, we need to call the 
ScanOracle (Algorithm n-times, where the gen- 
eration of the subsets costs n 2 , the oracle (that per- 
forms the simulations in parallel) is linear and the 
elimination of the duplicates is n 2 , so the cost of the 
ScanOracle algorithm is o(n 2 ). The cost of putting 
the agents in $ is a constant. So, the total cost of the 
algorithm is n 2 + n x 0(n 2 ) +C = 0(n 3 ). □ 

The computational cost of the non-deterministic 
method implemented in Algorithm [2] is the same of 
the deterministic algorithm. This can be puzzling for 
the reader, who can be expecting a lower cost, since 
the non-deterministic version is obtained from the de- 
terministic version by using an oracle. In particu- 
lar, the reduction by an oracle can be used to prove 
that a problem of polynomial complexity on deter- 
ministic machines can be solved in logarithmic time 
on non-deterministic machines. We made a different 
choice, for specific reasons. First of all, we employ 
the simulation step, that is not incorporated into the 
first part of the method, and thus the solution cannot 
be computed in a time lower than polynomial, in a 
non-deterministic fashion, even if we use, as we did, 
the oracle call. As a consequence of this choice, we 
could define, by using the same structure of the non- 
deterministic algorithm, a variant of the algorithm in 
which the solutions of the oracle are compared to each 
other, to choose the optimal one. 

We say that a blocking method, and thus the cor- 
responding algorithm, is sound if it does not allow a 
system to go in an insecure state when it starts from a 
secure state Sq. 

Theorem 3 The greedy blocking method, Algo- 
rithm^ is sound. 

Proof For the sake of contradiction, assume that the 
greedy blocking method brings a system in an inse- 
cure state. This means that the algorithm allowed a 

r, 
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Figure 1: A state graph for a system with two variables. 
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Figure 2: A state graph with secure and insecure states. 



insecure while S t is secure. But, by definition, our al- 
gorithm allows only transitions that bring the system 
in a secure state, thanks to the simulation. □ 

Let us now define the notion of a state graph. Re- 
call that every state of the system is defined by an as- 
signment of truth values to the variables, and a state is 
secure if it falsifies all security-related formulas. As 
a very simple example, in Figure Q] we give the state 
graph of a system with two variables {A,B}. 

In general, to denote the transitions executed by a 
system, we build a state graph as follows: every state 
is represented as a vertex of the graph, and every pair 
of vertices is connected by an edge when and only 
when the two edges differ by the truth value of one 
single variable, where the edge is labeled by the name 
of the agent that controls that variable. The resulting 
graph is indirected. In Figure |2 we give an example 
of such a graph, where we omit to specify all the val- 
ues of the variables for readability, but instead denote 
with gray vertices the insecure states and with white 
vertices the secure ones. 

We say that a blocking method, and thus the corre- 
sponding algorithm, is complete if every secure state 
can be reached from any secure state. To prove the 
completeness of the greedy blocking method, we pur- 
sue the following strategy: 

1 . First we prove that the state graph of the system is 
connected. 



2. We prove that the subgraph formed by the vertices 
representing secure states is connected when the 
security formulas can be written as a set of Horn 
clauses. 

3. We prove that every formula that we consider can 
be written as a disjunction of Horn clauses. 

4. We show that two secure states, whose security 
formulas can be written as Horn clauses, are con- 
nected if and only there is a path of secure states 
in the above mentioned subgraph that can be tra- 
versed by the algorithm. 

5. We show that the set of agents that have to be 
blocked, defined by a rewriting into Horn clauses 
of a security formula, is the union of agents that 
control variables occurring in one single Horn 
clause, and that can modify the value of the for- 
mula. 

6. We show that the set of agents blocked by the 
greedy algorithm is a superset of the set of agents 
that control variables occurring in one single Horn 
clause in any rewriting of the formula. 

In particular, we write H (<&,<&') to denote the set 
of the agents that control at least one variable of one 
Horn clause in one rewriting O'of the security formula 
<t>, in such a way that by changing the value of one of 
these variables the value of the security formula can 
pass from _L to T. 

Before we carry out this sequence of proof steps, 
let us observe a few simple facts that will be useful 
in the following. First of all, every secure state cor- 
responds to a formula, obtained as the conjunction of 
the literals representing the truth values of the vari- 
ables in that state. Since the single elements of the set 
of security formulas have to be false for the system to 
be secure, we can describe this situation directly by 
the set of secure states. Indeed, guaranteeing false- 
ness of each security formula corresponds to falsify- 
ing the disjunction of the logical expressions repre- 
senting the secure states. 

Lemma 1 The state graph is connected. 

Proof This follows straightforwardly by the defini- 
tion of state graph. □ 
It would be tempting to presume that not only the 
set of states is connected, but also the set of secure 
states. However, this is untrue. Consider namely the 
case in which the system has two variables, A and B, 
so that there are four states as shown in FigureQ] Sup- 
pose that the security formula is (-A AB) V (A A -ifi). 
The set of secure states is formed by the state in which 
both A and B are false and the state in which they are 
both true. Clearly the set of secure states is then dis- 
connected. 



Conversely, if the set of secure states is connected, 
the security formula can be written as a Horn clause 
(or a set of clauses, which is equivalent). To do so, 
we introduce the notion of Horn rewriting of a for- 
mula: a propositional formula is a Horn clause iff it 
can be written in Conjunctive Normal Form (i.e. as a 
conjunction of disjunctions of literals) in which ev- 
ery conjunct is formed by at most one positive literal 
(This is the standard notion of Horn clause, which we 
recall for preserving self-containedness.) It is well- 
know that every propositional formula can be written 
as a disjunction of Horn clauses. 

A Horn labeling X of the states of a system is an 
assignment of the system variables to one of the cor- 
responding literals. Whenever, in a Horn labeling, 
A,(v) = v for a variable v, the literal v will be consid- 
ered positive by that labeling, and consequently literal 
-iv will be considered negative. If X(v) = ->v, then the 
literal v will be considered negative, and consequently 
literal ->v will be considered positive. We henceforth 
generalize the notion of Horn clause, by stating that 
a formula is a Horn clause when there exists a Horn 
labeling for it that makes it a Horn clause. 

In the above example, the formula can be rewrit- 
ten, by applying the distributive property, as (A VB) A 
(-A A -iB) and there exists a Horn labeling that makes 
the formula a Horn clause: X(A) = -A and X(B) = B. 

We can now prove a property that will be useful in 
the following. 

Lemma 2 If the set of states that correspond to a se- 
curity formula is connected, then the security formula 
is a Horn clause. 

Proof As said above, the security formula <t> can be 
written as the disjunction of the conjunctions of lit- 
erals representing the interpretation of letter^] occur- 
ring in the formula itself for all the secure states. We 
henceforth say, when no confusion arises, that <t> is 
the disjunction of secure states. 

There are two methods to obtain an equivalent for- 
mula <t>' from <t>. We can use the distributive property 
or we can negate the formula obtained by considering 
the insecure states. 

• For the first approach, since the variables of the 
system all occur in each single state, the distribu- 
tion provides two types of subformulas in <!>', that 
are conjuncts of the security formula: complete 
conjuncts, obtained by the combination of all vari- 
ables, and incomplete conjuncts, obtained by the 
combination of a subset of the variables occurring 



5 A letter is interpreted in classical semantics. The truth 
value T is represented by the positive literal, whilst the 
value _L is represented by the negative literal. 



in the set of secure states. The incomplete con- 
juncts can occur only when some variables only 
occur in one form as a literal in the single dis- 
juncts of the formula <t>. 

• For the second method, consider the insecure 
states, which form the complement set of the se- 
cure states in the system. Clearly, the security for- 
mula can be written as the negation of the formula 
obtained as a disjunction of the insecure states. 
Since the negation of a disjunctive formula is a 
conjunctive formula, the obtained object is in con- 
junctive normal form. 

The new formula is a conjunction of disjunctions of 
literals, in which every literal is the disjunction of the 
complemented literals of an insecure state. 

We prove that, regardless of the number of in- 
volved variables, there is always a valid Horn labeling 
by means of a rewriting that is, depending on the num- 
ber of involved states, either based on the distributive 
property or on the negation of insecure states. In fact, 
given the number of states s in the set of secure states 
for a given formula <t> and the number n of variables 
of the system, we can have s < n + 1, s = n + 1 or 
s > n + 1. 

(s < n + 1) If the number of secure states is less than 
or equal to the number of variables, since this 
set has to be connected, by hypothesis, then the 
number of complete conjuncts cannot be greater 
than the number of variables. Therefore, there 
is always at least one incomplete conjunct in <t>' 
formed by one single literal v. If this happens, 
every time that the value of that literal is -iv, the 
security formula is false, so we can rewrite the for- 
mula as the conjunction of the single literal con- 
juncts, and consider, as Horn labeling, the one ob- 
tained by associating to the variables in the single 
literal conjuncts the negation of the literals occur- 
ring in those conjuncts. 

(s — n+l) If the number of disjuncts in <t>, namely 
the number of secure states, is exactly the num- 
ber n+l, two situations can occur, in principle. 
Either there are single literal conjuncts or there 
are not. Actually, due to the connectedness of the 
state graph, the former is impossible. If there are 
single literal conjuncts, then one variable always 
occurs in the same literal form in all the disjuncts 
of <t>. Consider one state in the set of secure states. 
To obtain all the other states in the set, we should 
be able to change one variable at a time and gener- 
ate in this way a new state. If the system contains 
k elements, then the above described process gen- 
erates k — 1 new states. But if there is one single 
literal conjunct in the rewriting <t>', then at least 



one variable cannot change ever during this pro- 
cess. Therefore, we can only generate n — 1 new 
states contrary to the hypothesis. When a set of 
states is formed in the way described above, then 
every single state always contains the same num- 
ber of positive and negative literals (with respect 
to the trivial Horn labeling), apart from one that 
contains one positive or one negative literal more 
than the others. This special state can be used to 
generate the correct Horn labeling for this set by 
inverting each literal in the A, function (positive lit- 
erals make the variables corresponding to negative 
literals and vice versa). 

(s > n+l) If the number of secure states is greater 
than n+l, then we can consider the rewriting of 
<t> obtained by negating the disjunction of insecure 
states, that we call 4>y. In other words, <t> = ^Oy. 
If we distribute the literals within <J>j/ and obtain 
the conjunctive normal form of <I>y , then applying 
the negation and again distributing the literals, we 
finally have a conjunctive normal form for <t> that, 
based on the properties of the graph, enjoys the 
same property of existence of single literal con- 
juncts discussed for the case s < n+ 1. Therefore 
the claim is proved. 

□ 

Let us now consider a generic set of states that is 
not connected. As we show in Figure[TJ this may any- 
how correspond to a valid Horn labeling. This, how- 
ever, does not occur for every security formula. Con- 
versely, every set of states can be written as the inter- 
section of connected sets of states. Therefore, given 
any security formula, we can represent it as the dis- 
junction of the Horn clauses that are obtained by the 
sets of connected states. 

The purpose of Algorithm[T]is to block the agents 
that apply for changing variables so to make true a 
critical formula. Since a critical formula can be made 
true by making true one of its disjuncts, Lemma|2]can 
be used directly to prove the following Lemma[3] 

More specifically, the greedy blocking works by 
blocking agents when they apply for the modifica- 
tion of the truth value of a variable, where the block- 
ing condition is: an agent cannot perform an action 
when this performance brings the system in an inse- 
cure state. The synchronization proposed by the algo- 
rithm is based on application time: the system simu- 
lates the result of performing all (up to the maximum) 
actions that agents applied for at that instant of time. 
The system denies the execution to those agents that 
modify variables involved in the transition of the sys- 
tem into an insecure state. Since this may correspond 
to more than one combination, the resulting blocked 
agent set may be larger than needed. We can assume, 



therefore, without loss of generality, that the algo- 
rithm blocks all the agents that applied for modifying 
variables that bring the system into an insecure state. 
This assumption is sufficient to employ fruitfully the 
generalization of Lemma [2] to generic formulas. Re- 
member that H (4>,<I>') denotes the set of the agents 
that control variables of one Horn clause in the rewrit- 
ing <J>'of <f> and bring the system into an insecure state. 

Lemma 3 If no agent in H (<t>, <!>') modifies variables 
occurring in 4>, and$> is false, then 4> is false after the 
modifications. 

Proof This is a direct consequence of the proof of 
Lemma[2] □ 
Since the agents blocked by the algorithm are all 
those that bring the system into an insecure state, then 
every agent controls variables that certainly occur in 
at least one disjunct of <t>. If we rewrite <t> as a disjunc- 
tion of Horn clauses (following the standard notion of 
formula rewriting into disjunction of Horn clauses) 

<J>' = <p\ V4>2 V... VO[, 

then, by definition of this rewriting, every variable 
controlled by a that occurs in <t>, occurs in at least 
one of these disjuncts. If an agent that controls one 
variable is blocked by our algorithm, then, by defini- 
tion of the simulation, at least one of the conjuncts in 
which the variable occurs in <t>' is true. This means 
that given any pair of secure states s and s', the al- 
gorithm never blocks an agent that brings the system 
directly from s to s'. The extension of this property to 
paths is proved in the following theorem. 

Theorem 4 The greedy blocking method, Algo- 
rithm^ is complete. 

Proof Consider two secure states s and s' , and one 
agent a that controls variables occurring in <t>. Sup- 
pose that s and s' are connected by a path of length 
k. If we can connect s to s' by a path of length k, 
then we can connect s to s" by a path of length k — 1 
and directly s" to s' . Suppose, by contradiction, that 
the greedy method blocks too many agents, so that 
the system would be able to move from s" to s' but 
not from s to s" . By Lemma [3] if one agent that 
is not blocked by the algorithm does not apply for 
changing one variable that occurs in one of the Horn 
clauses that appear in a rewriting of <t>, then the for- 
mula remains false. This would mean that at least 
one agent, able to perform the changes of variable 
values that would bring the system from s to s' ', con- 
trols at least one variable that does not occur in each 
possible rewriting of <t>. This leads to the conclusion 
that at least one variable is controlled by one agent 
a, blocked by the greedy method, since a is able to 
bring the system into an insecure state, from s" to 



s', the same variable is controlled by another agent 
a', that vice versa could move the system to an inse- 
cure state and occurs in one Horn disjunct of one of 
the possible rewritings of <t>, and brings the system 
from s to s" . As a consequence, one variable would 
be controlled by two agents, which is contrary to the 
definition of agent control of variables in the system 
definition adopted here. □ 
Soundness and completeness of the deterministic al- 
gorithm directly extend to the non-deterministic one. 

Theorem 5 The non-deterministic blocking method, 
Algorithm^ is sound. 

Proof The non-deterministic blocking method is an 
extension of the greedy blocking method of Algo- 
rithm[T]by means of an oracle. This means that every 
solution of Algorithm Q] is also a solution of Algo- 
rithm|2] □ 

Theorem 6 The non-deterministic blocking method, 
Algorithm^ is complete. 

Proof The same reasoning used to prove Theorem[5] 
applies for completeness. □ 
Let us call optimal a method that blocks the small- 
est sets of agents to ensure the security of the system. 
The greedy blocking method guarantees just one of 
the optimality properties, i.e. security, but it cannot 
guarantee to block the smallest sets of agents. We thus 
say that the greedy blocking method is a sub-optimal 
solution. 

What can further be proved is that the comparison 
of the solutions computed in the non-deterministic 
method generates an optimal solution. This is quite 
obvious, since the solutions computed are all the pos- 
sible combinations, and thus the best solution is in- 
cluded in this set. What the algorithm does is find the 
smallest set of agents that need to be blocked^ 

Theorem 7 Algorithm [2] computes an optimal solu- 
tion. 

We consider here the specific problem of blocking 
underhand attacks as the problem of keeping the se- 
curity formula false when agents apply for changing 
variables. The computational complexity of a prob- 
lem is defined as the cost of the best solution. In this 
case, we cannot claim that the solution is optimal and 
therefore we only have an upper bound result. 

Theorem 8 Blockage of underhand attacks is a poly- 
nomially solvable problem on deterministic machines. 

6 There may exist more than one solution with the small- 
est number of agents blocked. The approach of Algorithm 
[2] is to compare everything with everything, so the chosen 
solution is the last examined one. 



Proof Algorithm Q] is deterministic, sound and com- 
plete, and its cost is polynomial. □ 
Analogously, the next result is a consequence of 
the results about soundness, completeness and cost of 
Algorithm^ again in form of an upper bound. 

Theorem 9 Optimal blockage of underhand at- 
tacks is a polynomially solvable problem on non- 
deterministic machines. 



6 Conclusions 

In this work, we have dealt with multi-agent systems 
whose security properties depend on the values of sets 
of logical formulas (the critical formulas of the sys- 
tems). We assumed that the values of these formulas 
can change due to actions performed by the agents of 
the system, and that some attacks can be performed by 
malicious agents that are authorized within the system 
itself (in other terms, users of the system). These at- 
tacks conducted from inside are underhand, and we 
focus specifically on those attacks that are performed 
by groups of individuals that do not reveal their be- 
longing to such groups, that we call hidden coalitions. 

We have introduced a deterministic method, im- 
plemented by the greedy blocking algorithm, which 
prevents attacks to the system carried out by hidden 
coalitions formed by agents that are users of the sys- 
tem itself. The method based on this algorithm is 
sound and complete, but the algorithm is not optimal 
as it cannot block the smallest set of potentially dan- 
gerous agents. The method is thus extended to a non- 
deterministic version that can be used, in future inves- 
tigations, to identify optimal solutions and to study 
extensively the computational properties of the solu- 
tion, from both deterministic and non-deterministic 
sides. The method is obtained by introducing an ora- 
cle within the deterministic version, which makes the 
soundness and completeness results directly applica- 
ble to the non-deterministic version as well. 

The starting point of our approach to model 



multi-agent systems is Coalition Logic (JPauly, 2001 
|Pauly, 2002[ |Pauly and Parikh, 2003[ >, a coop- 
eration Logic that implements ideas of Game 
Theory. Another cooperation logic that works 
with coalitions is the Alternating-time Tempo- 
ral Logic (|Aluret al„ 1998a! |Aluretal„ 1998bl 



Walt her et al., 1997) l. A widely used logic, specif- 
ically thought for dealing with strategies and 
multi-age nt systems, is the Quantified Coalition 
Logic (|Agotnes et al., 2007| 
|Wooldridge and Dunne, 2004| ). 



Agotnes et al., 2008 



A specific exten- 
sion, also used for agents in multi-agent systems 
is CL-PC ( |van der Hoek and Wooldridge, 2005| 



Troquard et al., 2009) , and this is indeed the version 
of Coalition Logic that we started from. 

The notion of hidden coalition is a novelty, and 
more generally, to the best of our knowledge, no spe- 
cific investigation exists that deals with security in 
open systems by means of a notion of underhand at- 
tack. The system presented here is a multi-agent one, 
where we did not discuss how these coalitions are 
formed or the negotiations that can take place be- 
fore the creation of the coalitions (Sandholm, 2004; 
Kraus, 1997). For future work, it will be interesting to 
consider in more detail the non-monotonic aspects un- 
derlying the problem of underhand attacks by hidden 
coalitions, e.g. to formalize: the mental attitudes and 
properties of the intelligent agents that compose the 
system, how agents enter/exit/are banned from a sys- 
tem or enter/exit a hidden coalition, and the negotia- 
tions between the agents for establishing the common 
goal and synchronizing/organizing their actions. In 
this work, we give a way to protect the system, with- 
out making a distinction between the case in which 
the agents that make the attack are actual members 
of a coalition or not. If the system is equipped with 
explicit/implicit coalition test methods, this can make 
up a significant difference in terms of usefulness of 
our approach. 

A specific analysis of the computational properties 
of our blocking methods, in particular an analysis of 
worst, average, and practical cases, will be subject of 
future work. Results of lower bound for the blocking 
problem and the optimal blocking problem, and the 
computational cost of the optimal blocking problem 
on deterministic machines are in particular important 
aspects to be investigated. 
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